The new features of AbiPy v0.9.1

M. Giantomassi and the AbiPy group

10th international ABINIT developer workshop
20-22nd May 2019 - Louvain-la-Neuve, Belgium

---

• These slides have been generated using jupyter, nbconvert and revealjs
• The notebook can be downloaded from this github repo
• To install and configure the software, follow these installation instructions

Use the Space key to navigate through all slides.

What is AbiPy?

Python package for:

• Generating ABINIT input files automatically
• Post-processing output results (netcdf and text files)
• Interfacing ABINIT with external tools
• Creating and executing ab-initio ABINIT-specific workflows

Dependencies:

• Hard deps: numpy, scipy, pandas, matplotlib, netcdf4, plotly, sympy, pymatgen
• Soft deps: ipython, jupyter notebooks, phonopy, ASE.
• AbiPy can be interfaced with ASE atoms via pymatgen converters

Two different workflows infrastructures:

Internal implementation (abipy.flowtk Flows):

• ✅ Lightweigth, no database is required
• ✅ Designed for rapid prototyping and for testing advanced ABINIT capabilities
• ❌ No explicit support for high-throughput (HT) applications.

AbiFlows package (requires Fireworks and MongoDB database)

• ✅ HT-oriented: use database to store workflow status for further analysis
• ✅ High-level no-brainer API designed for HT applications
• ❌ New ABINIT features are first implemented/tested with flowtk Flows and then ported to AbiFlows.

NB: There's an ongoing effort to reimplement AbiFlows in terms of the atomate framewors

What's new at the level of the installation procedure?

• AbiPy now requires python >= 3.6
• conda packages are now provided by conda-forge
• AbiPy v1.0 will drop support for Abinit8

How to install AbiPy

Using conda (recommended):

    conda install abipy --channel conda-forge

Recipes for AbiPy and ABINIT are now supported by the conda-forge community 🎉

This means that:

• conda packages are automatically generated when new ABINIT/AbiPy releases are pushed to github
• the abiconda channel is deprecated. Please use the conda-forge versions

Since conda is not limited to python packages, one can install ABINIT in the same env with:

    conda install abinit -c conda-forge

Using pip and python wheels:

    pip install abipy --user

From the github repository (develop mode):

    git clone https://github.com/abinit/abipy.git
    cd abipy 
    python setup.py develop

---

For further info see http://abinit.github.io/abipy/installation.html

What's new at the level of the documentation?

• New AbiPy website based on Read-the-Docs theme
• Jupyter notebooks
• Gallery of matplotlib examples and flows
• abitutorial github repo with additional example

New AbiPy website based on Read-the-Docs theme

%embed https://abinit.github.io/abipy/index.html

Sphinx-gallery with docs, figures and links to download py scripts, notebooks or run the example via binder.

%embed https://abinit.github.io/abipy/gallery/plot_phbands_nkpt_tsmear.html#sphx-glr-gallery-plot-phbands-nkpt-tsmear-py

%embed https://abinit.github.io/abipy/gallery/index.html

%embed https://abinit.github.io/abipy/flow_gallery/index.html

%embed https://nbviewer.jupyter.org/github/abinit/abitutorials/blob/master/abitutorials/index.ipynb

What's new at the level of workflow machinery?

For the DFPT part:

• Python converters DDB $\;\rightleftarrows \;$ phonopy / tdep

• New Flows to compute:

  • elastic and piezoelectric tensors with DFPT (clamped/relaxed atoms)
  • effective masses with DFPT or finite differences

For the e-ph part:

• New flows and post-processing tools for:

  • e-ph self-energy
  • transport properties
  • e-ph matrix elements and scattering potentials

DDB converters

(contributed by G. Petretto)

• Based on py script by A. Romero. ...

• Applications:

  • Interface AbiPy with phonopy
  • Connect ABINT DPT with other requiring phonony (e.g. anharmonic calculations hipive)

Example: obtain irreps from DDB using phonopy

ElasticWork

Flow to run DFPT calculations to compute:

• the rigid-atom elastic tensor
• the rigid-atom piezoelectric tensor (insulators only)
• the internal strain tensor
• the atomic relaxation corrections to the elastic and piezoelectric tensor

scf_input = make_scf_input() # Build input for GS calculation

elast_work = flowtk.ElasticWork.from_scf_input(scf_input, 
                                               with_relaxed_ion=True, 
                                               with_piezo=True)

Effective masses with DFPT

Flow to run DFPT calculations to compute:

• the rigid-atom elastic tensor
• the rigid-atom piezoelectric tensor (insulators only)

https://abinit.github.io/abipy/flow_gallery/run_effmass_dfpt.html#sphx-glr-flow-gallery-run-effmass-dfpt-py

Flow to compute e-ph matrix elements along an arbitrary q-path

Flow to run DFPT calculations to compute:

• the rigid-atom elastic tensor
• the rigid-atom piezoelectric tensor (insulators only)

This example shows how to compute the e-ph matrix elements in AlAs along a q-path with AbiPy flows. The final results are stored in the GKQ.nc file (one file for q-point) in the outdata of each task.

# Build input for GS calculation on a 2x2x2 k-mesh
scf_input = make_scf_input(ngkpt=(2, 2, 2))

# q-mesh for phonons
ngqpt = (2, 2, 2)

# Create flow to compute all the independent atomic perturbations
# Use ndivsm = 0 to pass an explicit list of q-points.
# If ndivsm > 0, qpath_list is interpreted as a list of boundaries for the q-path
qpath_list = [[0.0, 0.0, 0.0], [0.01, 0, 0], [0.1, 0, 0],
              [0.24, 0, 0], [0.3, 0, 0], [0.45, 0, 0], [0.5, 0.0, 0.0]]

from abipy.flowtk.eph_flows import GkqPathFlow
flow = GkqPathFlow.from_scf_input("flow_dir", scf_input,
                                  ngqpt, qpath_list, ndivsm=0, with_becs=True,
                                  ddk_tolerance={"tolwfr": 1e-8})

https://abinit.github.io/abipy/flow_gallery/run_gkq.html#sphx-glr-flow-gallery-run-gkq-py

What's new at the level of the post-processing tools?

• New plotting tools based on plotly
• GUIs and dashboards based on panel and bokeh

Matplotlib vs Plotly

---

• ✅ Publication quality figures.
• ✅ Flexible python API able to produce rather advanced plots
• ❌ Plots are difficult to customize without changing the python code
• ❌ Plots lacks interactivity and integration with HTML.

---

• ✅ Interactive plots + chart editor GUI to customize the figure
• ✅ Play well with HTML (plotly us written in js with python bindings)
• ❌ Open source project but not all the features are available in the free plan
• ❌ Requires browser (this may represent an issue on some HPC centers)

AbiPy plots with matplotlib

gsr = abiopen("si_nscf_GSR.nc")
gsr.ebands.plot(with_gaps=True);

AbiPy plots with plotly

(contributed by Y. He and MG)

• Convention: object.plot becomes object.plotly

gsr.ebands.plotly(with_gaps=True);

To update the plotly figure to the chart studio server, use:

gsr.ebands.plotly(with_gaps=True, chart_studio=True);

Requires chart-studio account and API token in $HOME/.pmgrc.yaml

%embed https://chart-studio.plotly.com/~gmatteo/214

Interactive 3d plots with plotly:

gsr.ebands.kpoints.plotly();

The matplotlib version embedded in HTML:

gsr.ebands.kpoints.plot();

Integrating Abipy with web-based technologies

What is Panel?

• Panel provides tools for composing widgets, plots, tables into apps and dashboards
• It relies on the client-server model where:
  • the client is the web browser running HTML/CSS/JS code.
  • the server communicates with the client, executes py code and sends the results back to the client
• In a nutshell, panel keeps the browser and python in synch e.g. the user clicks a button in the GUI and the signal to python.

Why panel?

• Panel works with visualizations from Bokeh, Plotly, Matplotlib, HoloViews, and many other Python plotting libraries,
• Panel works equally well in Jupyter Notebooks

Thanks to this infrastructure, developers can use python to solve the scientific problem using the pydata ecosystem and the packages developed by the ab-initio community.

Pros and cons of the client-server model

Advantages:

• Client does not need to install the scientific software stack (when running on different machines)
• Can take advantage of modern HTML technologies to present the results in the browser
• Can implement web apps that allows the user to upload the data and analyze the results withouth having to install python software.

Disavantages

• Round trip delay if client != host and slow connection
• Upoloadng a 1Gb file to the remote server just because you don't want to install software on the localhost is a very bad idea.
• OK for relatively small files (< 1Gb) but this approach is not designed to handle big data.
• Not all the HPC centers provide specialized nodes to post-process the results inside a web browser/notebook.

AbiPy panels inside jupyter notebooks

To build a panel GUI inside the notebook, use:

#gsr = abiopen("si_nscf_GSR.nc")
#abilab.abipanel();
#gsr.get_panel()

ddb = abilab.abiopen("ZnSe_hex_qpt_DDB")
abilab.abipanel();
ddb.get_panel()

Other files are supported as well

ddb.structure.get_panel()

Creating dashboards from the CLI

• Using panel inside notebooks is great if you need both the flexibility of the python language and the easiness of the GUIs to improve your productivity.
• In some cases, however, we just want to create a dahhboard with widgets to interact with the data.

To create a panel dashboard associated to the DDB file, use:

abiopen.py ZnSe_hex_qpt_DDB --panel  # or -pn if you prefer the short version.

To produce a predefined set of matplotlib figures, use:

abiopen.py mgb2_kpath_FATBANDS.nc --expose --seaborn

abiopen.py mgb2_kpath_FATBANDS.nc --expose-web # -ewb

abiopen.py mgb2_kpath_FATBANDS.nc --plotly # -ply

Replace --expose with --notebook to generate a jupyter notebook with predefined python code

Creating dashboards without widgets with abiview

to get a quick look at the results

abiview.py ddb ZnSe_hex_qpt_DDB --panel  # or -pn if you prefer the short version.

## Documentation of new features available at:

%embed https://abinit.github.io/abipy/graphical_interface.html

Need to call anaddb to compute and visualize ph-bands and DOS from DDB?

abiview.py ddb ZnSe_hex_qpt_DDB --seaborn

To compare multiple structures:

!abicomp.py structure *.cif si_nscf_GSR.nc `find . -name *_DDB`

zsh:1: no matches found: *.cif

Note shell wildcard characters and Unix find inside backticks (bash rocks!)

#plotter = abilab.ElectronBandsPlotter()
#plotter.add_ebands(label="BZ sampling", bands="si_scf_GSR.nc")
#plotter.add_ebands(label="k-path", bands="si_nscf_GSR.nc")
#plotter.gridplot(with_gaps=True);

Using ElectronBandsPlotter to visualize multiple band structures:

abicomp.py ebands *_GSR.nc

Bash is handy but python is more flexible

Let's use the DdbRobot to compare phonons obtained with different ${\bf k}$-meshes and smearing values:

paths = [
    "mgb2_888k_0.01tsmear_DDB",
    "mgb2_888k_0.04tsmear_DDB",
    "mgb2_121212k_0.01tsmear_DDB",
    "mgb2_121212k_0.04tsmear_DDB",
]

paths = [os.path.join(abidata.dirpath, "refs", "mgb2_phonons_nkpt_tsmear", f) 
         for f in paths]

robot = abilab.DdbRobot()
for i, path in enumerate(paths):
    robot.add_file(path, path)

# Define function to change labels:
func = lambda ddb: "nkpt: %s, tsmear: %.2f" % (
    ddb.header["nkpt"], ddb.header["tsmear"])

robot.remap_labels(func)
robot

0. nkpt: 256, tsmear: 0.01
1. nkpt: 256, tsmear: 0.04
2. nkpt: 864, tsmear: 0.01
3. nkpt: 864, tsmear: 0.04

Now we can build a dataframe with the most important parameters:

robot.get_params_dataframe()

	nkpt	nsppol	ecut	tsmear	occopt	ixc	nband	usepaw
nkpt: 256, tsmear: 0.01	256	1	35.0	0.01	4	1	8	0
nkpt: 256, tsmear: 0.04	256	1	35.0	0.04	4	1	8	0
nkpt: 864, tsmear: 0.01	864	1	35.0	0.01	4	1	8	0
nkpt: 864, tsmear: 0.04	864	1	35.0	0.04	4	1	8	0

and check that all DDBs have been computed with the same crystalline structure:

robot.get_lattice_dataframe()

	formula	natom	alpha	beta	gamma	a	b	c	volume	abispg_num	spglib_symb	spglib_num	spglib_lattice_type
nkpt: 256, tsmear: 0.01	Mg1 B2	3	90.0	90.0	120.0	3.086	3.086	3.523	29.055953	0	P6/mmm	191	hexagonal
nkpt: 256, tsmear: 0.04	Mg1 B2	3	90.0	90.0	120.0	3.086	3.086	3.523	29.055953	0	P6/mmm	191	hexagonal
nkpt: 864, tsmear: 0.01	Mg1 B2	3	90.0	90.0	120.0	3.086	3.086	3.523	29.055953	0	P6/mmm	191	hexagonal
nkpt: 864, tsmear: 0.04	Mg1 B2	3	90.0	90.0	120.0	3.086	3.086	3.523	29.055953	0	P6/mmm	191	hexagonal

To analyze the effect of k-sampling/smearing on the vibrational properties:

# Invoke anaddb and store results
r = robot.anaget_phonon_plotters(nqsmall=2)

r.phbands_plotter.gridplot_with_hue("tsmear", with_dos=True);

#multi = abilab.ebands_input(structure="si.cif", 
#                            pseudos="14si.pspnc",
#                            ecut=8, 
#                            spin_mode="unpolarized", 
#                            smearing=None, 
#                            dos_kppa=5000)

#multi.get_vars_dataframe("kptopt", "iscf", "ngkpt")

Future developments

Post-processing tools

• Support panel for more netcdf files
• More converters and interfaces for third-party applications
• Integrate AbiPy with jupyterlab to create a flexible graphical enviroment for Abinit exposing (part) of the python API
• Explore web-based technologies for data analysis and visualization (plotly, dash)
• Develop toolkit to build web apps powered by AbiPy and pymatgen to disseminate scientific results.

Continuous Integration

• Use AbiPy programmatic interface to implement:

  • Validation of parallel algorithms for np in range(1, N)
  • Stress testing
  • Benchmarks